[RNE Rewrite] test(ts): add API tests for the TypeScript surface - #1355
Open
msluszniak wants to merge 8 commits into
Open
[RNE Rewrite] test(ts): add API tests for the TypeScript surface#1355msluszniak wants to merge 8 commits into
msluszniak wants to merge 8 commits into
Conversation
msluszniak
marked this pull request as draft
August 7, 2026 14:36
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 10, 2026 06:29
49728d3 to
4755b35
Compare
Every path through `src/` bottoms out in `__rnexecutorch_jsi__`, so stubbing those calls per test would only ever assert against the stub. Instead of stubbing, implement the native contract in JavaScript: typed-array-backed tensors with the real byte semantics, JS implementations of the math/cv/speech operators, and a `loadModel` that serves a program the test describes. Task pipelines therefore run end to end. Alongside it, an in-memory blob-util mock with a programmable server (status, body, Range support, and a gate to hold a download open) and a worklets mock that runs worklets inline. Native memory is not garbage collected, so the setup file asserts after every test that nothing allocated through the fake was left undisposed; each pipeline suite gets disposal coverage for free.
- `core/`: tensor byte semantics and copy windows, model execution and disposal, `wrapAsync` error propagation, and the spec matcher in depth — symbol binding, variant selection, runtime-constraint matching, and the authoring errors that fire before matching starts. - `fetcher/`: caching, forced re-download, byte-weighted progress, HTTP and transport failures, cancellation, requests shared between concurrent callers, iOS partial-file resume, and the Android DownloadManager backend. Telemetry gets its own suite, including the locale parsing that must not read a language-only tag as a country. - `extensions/`: box and point scaling under both resize modes, box decoding, and the seeded generators.
One suite per pipeline: which model signatures it accepts, that a mismatch is rejected with a message naming the mismatch, the postprocessing that is the pipeline's own work (softmax ordering, NMS suppression, argmax colormaps, sigmoid grayscale masks, coordinate scaling back through letterboxing), every option and per-call override, and that `dispose()` releases everything. The pipelines whose behavior depends on real model weights — Whisper, VAD, SDXS, keypoints, instance segmentation — get schema acceptance, rejection and full disposal instead, including Whisper's nested tokenizer and VAD pipeline. `tasks/constructionFailure.test.ts` records a leak the suites surfaced: a `create<Task>` factory that throws after `loadModel` abandons the native model, and the caller never receives a `dispose` to release it. The tests assert the current behavior so it flips loudly once fixed. Hook suites cover the lifecycle apps depend on: disposal on unmount and on config change, the create-after-unmount race, config identity by value, download cache hits, preventLoad, and errors surfaced through the shared field.
- A snapshot of every export, so an addition, rename or removal shows up in the diff of the pull request that causes it rather than in a user's app. - Registry rules that only fail on a device otherwise: https URLs on the software-mansion org, a pinned revision, the `modelname_backend_precision.pte` naming contract, a folder matching the backend suffix, and a default alias structurally identical to one of its own variants. - Label-array invariants, including the ImageNet duplicates that must stay because the array mirrors the model's output vocabulary. - Source-level conventions, parsed with the TypeScript compiler: the `'worklet'` directive on every JSI wrapper, and the core/extensions and hooks/native import boundaries from the architecture guide.
Adds an `api-tests` job to CI — no native libraries, no simulator, no `.pte`, so the existing TypeScript-only setup action is all it needs. Adds an `add-api-tests` skill covering the fake runtime, the helpers, what to cover for a new pipeline or hook, and the leak-checking contract; wires it into the skills index, the architecture guide and the maintenance list; and adds the test step to the verify-and-build workflow and checklist. Also refreshes the hook example in `add-task-pipeline`, which still showed the `localPath` shape from before the resource fetcher landed.
Rebasing onto the supertonic TTS pipeline (#1317) surfaced two gaps the suites themselves reported: - The registry walk assumed every task config names a single `modelPath`. Supertonic assembles four `.pte` files under `modelPaths`, so its category looked empty and its variant group went unchecked. Recognize both shapes, and compare variants on the whole set of files they name. - The export snapshots record the new TTS surface. Additions only — no removal or rename.
Adapts the suites to what landed on `rne-rewrite` since they were written: hooks report absence as `undefined` rather than `null`, aborts surface as a coded `DOWNLOAD_ABORTED` rather than an `AbortError` class, `schema.constr` is now `schema.constraint` with `equality` in place of `eq`, the CV ops moved to singular module names, `bool` joined `DType`, and the iOS download path reads its status through blob-util's `stateChange`, which the mock now serves. Two registry conventions were restated rather than relaxed. Variant groups name their default with an explicit `DEFAULT` key instead of spreading it, and a default may now live further down the tree (a YOLO26 family defaults through a scale and an input size), so the rule is that the default has to be a config the group actually offers. Model files are still required to declare their backend and precision, but Kokoro nests a variant folder under the backend one and publishes its grapheme-to-phoneme models per language, so the backend is looked for anywhere in the path and the untagged files are pinned to that one shape rather than left to opt out silently. The three source fixes the suites originally carried are all upstream already, so the commit that made them is gone.
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 26, 2026 18:58
4755b35 to
499a2ad
Compare
`rne-rewrite` gained the privacy filter, the LLM chat session, Kokoro TTS, PaddleOCR and the coded error type since these suites were written, and none of them had a suite. Only the weights are out of scope, so most of this runs end to end. The privacy filter's logits come from a scripted `execute`, and everything above them is driven for real: the BIOES grammar, the Viterbi decode, the sliding window and its overlap policy, and the static-versus-dynamic export handling. The chat session runs against a runner whose KV cache is a token position, which is enough to check that a second turn prefills only what is new, that a tool loop feeds results back, and that a failed turn leaves the session usable. Kokoro gets its argument validation, chunking and streaming; OCR gets its quad decode, CTC collapse, confidence filter and reading order over a probability map the test paints. That needed the fake runtime to grow the parts those pipelines reach: `llm.createLLMRunner`, `speech.createPhonemizer`, `math.gather`, `cv.extractDbnetTextQuads`, `cv.rectifyQuad` and `fs.readFile`. The two CV decoders implement their documented contract over axis-aligned regions rather than tracing contours and warping perspective, which is the same answer for the rectangles a pipeline test draws and keeps every threshold the caller passes load-bearing. The leak check now also covers LLM runners and phonemizers, so the pipelines that own them get disposal coverage the same way.
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 26, 2026 19:06
499a2ad to
5e6e7e9
Compare
12 tasks
msluszniak
marked this pull request as ready for review
August 26, 2026 20:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds TS API tests:
Also adds necessary skills.
To find details about this approach please follow:
__tests__/README.md.No source changes: the three contract violations the suites originally surfaced (
SpecMatch.dim, the missing'worklet'directive ongetRegisteredBackends, andrandomNormal's millisecond-resolution default seed) have all since been fixed onrne-rewrite, sosrc/is untouched by this PR.During the testing I spotted a problem that is addressed in #1386.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
yarn workspace react-native-executorch test yarn typecheck yarn lintExpected: 28 suites, 3072 tests, 4 snapshots, 0 skipped; typecheck and lint clean.
yarn preparestill emits onlysrc/intolib/.Screenshots
N/A
Related issues
Closes #1352.
Checklist
Additional notes
Layout
core/fetcher/tasks/hooks/useModel,useResourceDownload, and the task hooks end to endextensions/api/support/Deliberately not covered: the numerical behavior of the native operators (that is
cpp/tests/, and duplicating it here would only test the fake); the parts of a pipeline whose behavior depends on real weights: Whisper's decode loop, the VAD rolling window and the SDXS diffusion step get schema acceptance, rejection and full disposal instead; and the worklet thread hop, since worklets run inline. The'worklet'directive convention that makes that hop possible is enforced by parsingsrc/with the TypeScript compiler.That line is drawn per pipeline, not per suite. The privacy filter's logits are weights but its BIOES decode and sliding window are not, so they run end to end; the LLM's generation belongs to the native runner but the chat session's history, KV cache bookkeeping and tool loop are covered against a scripted one; Kokoro's waveform is weights but its argument validation, chunking and streaming are not; PaddleOCR's probability map is weights but the quad decode, CTC collapse and reading order run over a map the test paints. The fake runtime grew
llm.createLLMRunner,speech.createPhonemizer,math.gather,cv.extractDbnetTextQuads,cv.rectifyQuadandfs.readFileto make that possible.